Systems, methods and devices for multispectral imaging and non-linear filtering of vector valued data

ABSTRACT

An apparatus and method for multispectral imaging comprising an array of filters in a mosaic pattern, a sensor array and an acquisition and processing module. The sensor array being disposed to receive an image that has been filtered by the array of filters. The acquisition and processing module processes the output of the sensor array (or mosaic acquired data) to provide a processed image. The acquisition and processing module processes the mosaic acquired data by performing first interpolation on the mosaic acquired data by the sensor array to provide a first approximation and performing second interpolation on the values of the first approximation to provide a second approximation.

RELATED APPLICATION

This application claims priority benefit under Title 35 U.S.C. § 119(e)of provisional patent application No. 60/719,004, filed Sep. 21, 2005,which is incorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

The present invention relates to devices and methods for enhancedimaging, and more particularly, in part, to multi-spectral imaging, bymosaics and arrays of sensors designed to measure a filtered version ofthe light reaching them, and to methodologies for calibrating theresponse and processing the output of such sensors and arrays.

There is a vast amount of technology that comprises the state of the artin advanced imaging and multispectral imaging. Some background may befound, for example, in a patent to some of the present inventors: U.S.Pat. No. 6,859,275, “System and Method for Encoded Spatio-SpectralInformation Processing”, Fateley et al., which is incorporated herein byreference in its entirety.

Spectral imaging systems typically have dynamic components such asmoving parts or adaptive parts such as tunable filters, in order toaccomplish the acquisition of spectral image information. These dynamiccomponents are typically expensive and require complex calibration andprocessing.

Color cameras typically measure only three channels of color information(such as red, green and blue). As such they are limited in their use asspectral imagers. Also, color cameras typically sample colors in aso-called mosaic pattern. This means that each pixel only measures onecolor, with the three different color measurements happening at threedifferent pixels—typically in a repeating pattern such as but notlimited to Bayer patterns. See U.S. Pat. No. 3,971,065 to Bayer, “ColorImaging Array”, which is incorporated herein by reference in itsentirety.

Therefore ordinary color cameras are limited in their capacity asadvanced or spectral imaging devices for at least two reasons. Firstbecause of the limited amount of spectral information that they provide(only 3 colors). Second because the full color information is notmeasured per-pixel but only by mosaic. In the U.S. Pat. No. 3,971,065,Bayer states that “relative image sampling rates, by color, are ineffect adjusted respective of the characteristics of human visualresponse.” While this is of some use for images to be viewed by humans,there is a need for more general systems, for example for scientific andengineering applications wherein human visual response is not the systemresponse of interest. Even when images are to be viewed by humans, thereis a need for an improved system for at least the following reasons.Standard techniques are available to model the missing information for amosaic sensor, and these comprise various kinds of interpolation. Suchinterpolations are used to “guess” or “fill in” the missing/non-measuredcolor information. However, these guesses can be wrong and the filled indata may, for example, appear blurry as a result. In particular, at anedge or boundary, simple interpolation will yield a result that can befar from correct. Consequently there is a need for improved methods forinterpolation of missing sampled data from mosaic cameras.

Beyond spectral imaging systems, advanced imaging devices in general areneeded for a variety of applications. Such advanced imaging systemsinclude but are not limited to high resolution imaging systems, widefield-of-view imaging systems, artifact correcting imaging systems, andhigh dynamic range imaging systems, to name a few. In both design anduse, state of the art imaging systems require tradeoffs between factorsincluding but not limited to resolution, field-of-view, contrast, imagequality, color, spectral detail and cost. Consequently there is and willalways be a need for improved imaging systems that allow for betterimage quality of these various kinds, or, put differently, that enablebetter choices to be made in, for example, the tradeoffs just described.

The state of the art in advanced imaging includes mosaic imaging forenhanced imaging including multispectral imaging and improvedresolution, field of view and other parameters. Examples of existingsystems for such may be found in the paper “Generalized Mosaicing”, bySchechner and Nayar, Eighth International Conference on Computer Vision(ICCV'01), 2001, Volume 1, pp. 17-24, which is incorporated byreference. In that paper, mosaic imaging systems are taught thataccomplish multispectral imaging, and other related designs for improvedresolution, field of view and other parameters. Briefly these goals areaccomplished by providing a mosaic of sensors with spatially variedproperties, so that as a camera is scanned over a scene, multiplemeasurements of each scene point are obtained under different opticalsettings. While this approach accomplishes the stated goals, it requiresscanning of a scene to produce the desired information. This scanningrequires registration from one frame to the next, to accomplish the datafusion. However, there is information in even a single frame of imagedata, and it is desirable to have systems that extract as much of thisinformation as possible. It is also desirable to have systems that,while capable of exploiting the extra information provided by scanning ascene multiple times, do not require such scanning.

In summary there is a need for devices, methods and systems for spectralimaging and advanced imaging with improved performance in one or more ofimage quality, contrast, resolution, field-of-view, color, spectraldetail and cost, and for improved methods for interpolation of missingsampled data from mosaic imaging systems.

OBJECT AND SUMMARY OF THE INVENTION

Therefore, it is an object of the present invention in one aspect, toprovide devices, methods and systems for spectral imaging by means of acamera comprising a mosaic pattern of spectral filters, together withthe necessary algorithms to substantially fill in the missing sampleddata from the mosaic information that the camera collects.

In accordance with an exemplary embodiment of the present invention, amosaic sensor array that measures a variety of optical properties of thepixels of the scene, and then applying an adaptive interpolation to fillin the missing data in accordance with the techniques disclosed herein.In doing so, the present invention allows for the substantial recoveryof a full set of distributed and possibly weighted bands of spectralresponsivity measures from a limited sampling of spectral bands withinthe spectral range of interest, by the methods disclosed herein. Theadaptive interpolation methods, described in detail elsewhere herein,are comprised of non-linear interpolation techniques wherein, for eachpixel, measured data at and near the pixel is used to create an adaptiveinterpolation filter for that pixel by mathematical inference. Thefilter is applied and the missing values are thereby substantiallyreconstructed.

Another object of the present invention is to provide devices, methodsand systems for advanced imaging by means of a camera comprising amosaic pattern of spatial or other optical filters, together with thenecessary algorithms to substantially fill in the missing sampled datafrom the mosaic information that the camera collects, and to reconstructan improved image wherein the improvement comprises one or more ofresolution, contrast, field of view, image quality and spectralinformation. In accordance with an exemplary embodiment of the presentinvention, this can be accomplished as disclosed in more detailelsewhere herein, in a way similar to the spectral object justdescribed—by providing a mosaic sensor array that measures a variety ofoptical properties of the pixels of the scene, and then applying anadaptive interpolation to fill in the missing data in accordance withthe techniques disclosed herein.

For exemplary illustrative purpose only, the present invention isdescribed herein in the context of spectral imaging. A key aspect isthat spectral information is vector information. That is, for each pixel(picture element) in an image, the spectral information is comprised ofa multiplicity of numbers associated with that pixel in a predeterminedway. It is important in understanding the scope of the presentinvention, to understand that other imaging vector information can beprocessed in the same way, and that the spectral imaging examplesdisclosed in detail herein are merely examples of the more generalinvention. Therefore when examples and embodiments are disclosed hereinin the context of spectral imaging, the use of spectral information isintended to be illustrative and not limiting, and these other advancedimaging embodiments are to be taken to be within the scope of thepresent invention.

A further object of the present invention is to provide improved imagingby modeling the improvement as the estimation of a vector for each pixelof the image being acquired, such that the vector field in questioncomprises information needed for the desired improvement. One exampleembodiment accomplishes improved image resolution as so described. Ahigh resolution image may be modeled as a vector-valued low-resolutionimage. For example, a 512×512 image can be modeled as a 256×256 image,wherein each pixel is a 4 dimensional vector comprised of the 4 valuesof the “sub-pixels” of the high-resolution image. Specifically, forexample, the vector associated with the first pixel in the first row ofthe low resolution image is comprised of the values of the pixels 1 and2 of the first row of the high-resolution image, together with pixels 1and 2 of the second row of the high-resolution image. In this way, ifone can acquire or substantially reconstruct the relevant 4 dimensionalvector of information for each pixel in the low-resolution image, onehas then reconstructed the high resolution image. As disclosed herein,an embodiment of the present invention achieves high resolution imagingby filtering each pixel, in a mosaic pattern, with a suite of filtersthat measure aspects of the high resolution image, and applying thealgorithms disclosed herein to the output of this sensor mosaic tosubstantially reconstruct the high-resolution image.

A device embodiment of the present invention comprises a mosaic ofsensors similar to a two-dimensional CCD wherein the detector elementsare each designed to measure a filtered version of the light spectrumreaching them. In such an embodiment, by virtue of the filters in or onthe detector elements, each pixel of the device measures a specific,predetermined function of the spatio-spectral data of the scene beingimaged. In an embodiment, the filters are taken from a predetermined setof filters, such set having more than one distinct element and typicallysubstantially fewer distinct elements than the total number of sensorsin the device so that many detectors in the device have the same filter.These sensor arrays are more complex, but are similar to conventionalRed, Green, Blue (RGB) cameras where the detector elements or individualsensors are organized into discrete groups of red green or blue sensors.In the embodiments being described, then, the sensors are divided intoclasses of filtered sensors, each class having substantially the samefilter function. An embodiment of the present invention, wherein thefilters are spectral filters, comprises distributing each class ofspectral sensors in a spatially uniform way across the array. Forexample in an application of uniform distribution we could have 9different spectral bands sensors in each 3×3 group of adjacent sensors,say by repeating periodically the first group of 3×3. Although we use aspatially uniform distribution for this example we do not limit thisdisclosure to uniform distributions of sensors or detector elements withsame or similar spectral responsivity. In applications where a uniformdistribution is not required or desired a weighted distribution ofspectral response functions can be implemented. For example we couldhave 9 spectral bands of responsivity with a multiplicity of one or morebands in the array as desired or needed.

Various embodiments of the present invention correspond with a varietyof configurations of sensor filters. An embodiment comprises dividing apre-defined spectral band such as the visible spectral region of 400 nmto 700 nm into 9 spectral bands. Another embodiment comprises dividingin a non-contiguous non-equispaced way the silicon spectral responseband of 280 nm-1100 nm in order to optimize the ability to determinespecific or generalized chemistries as needed. Further details andadditional examples include but are not limited to the assignment ofHadamard encoded spectral filters to the various pixels within a set ofpixels; the assignment of Fourier encoded spectral filters to thevarious pixels within a set of pixels; the assignment of randomly orpseudo-randomly encoded spectral filters to the various pixels within aset of pixels (particularly if the pseudo-random codes are selected insuch a way that the family of codes is well conditioned); and others.Techniques for creating these filters include, but are not limited to,depositions of layers and/or films of wavelength dependenttransmissivity/opacity, holographic films, quantum dots, photoniccrystals and other special materials.

One embodiment comprises dividing the red region spectral band intothree sub-bands by rejecting or passing a different ⅓ for each of threesensors, with a similar scheme of division implemented in the remainingblue and green spectral bands. Embodiments of the invention comprisemathematically encoding the filters on each sensor, and/or weightingthem in a spectrometrically significant way that enables recovery ofdesired classes of spectral features that, for example, relate tospecific materials or chemistries of interest in a qualitative and orquantitative manner.

The multispectral camera systems disclosed herein have use in a widevariety of applications, including but not limited to automated targetrecognition, detection and tracking of explosives, biohazards and otherthreats.

It should be noted that different embodiments of the invention mayincorporate different combinations of the foregoing, and that theinvention should not be construed as limited to embodiments that includeall of the different elements.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood and appreciated more fully fromthe following detailed description, taken in conjunction with thedrawings in which:

FIG. 1 shows a filter mosaic pattern for a mosaic filter imaging systemin accordance with the prior art and, when combined with the processingdisclosed herein, appropriate for use with an exemplary embodiment ofthe present invention;

FIG. 2 shows another filter mosaic pattern with a 3 by 3 array mosaic;

FIG. 3 shows another filter mosaic pattern corresponding to the Bayerpattern;

FIG. 4 shows another filter mosaic pattern with a 4 by 4 array mosaic;

FIG. 5 shows a system diagram for a device in accordance with anexemplary embodiment of the present invention;

FIG. 6 shows a flowchart for a process in accordance with an exemplaryembodiment of the present invention, for filling in data from the outputof a mosaic sensor in accordance an exemplary embodiment of the presentinvention; and

FIG. 7 shows a flowchart for a process in accordance with an exemplaryembodiment of the present invention, for filling in data from the outputof a mosaic sensor in accordance with an exemplary embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Turning now to FIG. 1, a filter pattern is shown comprising, in thiscase, 4 filters 110, 120, 130 and 140, which repeat in a 2 by 2 mosaicpattern. FIG. 2, FIG. 3 and FIG. 4 show additional mosaic patterns. FIG.2 shows a 3 by 3 repeating mosaic pattern comprising the filters labeled210, 220, 230, 240, 250, 260, 270, 280 and 290 which appear in the 3 by3 pattern, and then repeat over the mosaic. In the figure, a pair ofrepetitions is shows vertically and horizontally. In practice many morerepetitions will occur, so as to cover the face of a sensor array asdescribed herein. FIG. 4 shows a 4 by 4 repeating mosaic patterncomprising filters 410, 415, 420, 425, 430, 435, 440, 445, 450, 455,460, 465, 470, 475, 480, and 485. The filter patterns shown areconsistent with the prior art and, when combined with the processingdisclosed herein, are appropriate for use with an embodiment of thepresent invention. FIG. 3 shows a mosaic corresponding to the Bayerpattern. For the present invention, essentially any predetermined filterpattern can be used, and the ones shown in FIGS. 1, 2, 3 and 4 aresimply examples.

The filters such as 110, 120, 130 and 140, 210-290, and 410-485 can bespectral band-pass filters, or more complicated coded or weightedspectral filters, and/or other filters including but not limited tospatial filters, custom lenses or lenselets, and spatio-spectralfilters. Options for spectral encoding include but are not limited tospectral Hadamard codes. For example each pixel in the 4 by 4 array ofsensor filter elements shown in FIG. 4 could measure eight bands among16 possibilities. The bands can be picked to cover uniformly ornon-uniformly the whole spectral range, as is standard in the art ofHadamard multiplexing. In this way, each individual band can berecovered from the knowledge of all 16 multiplexed measurements. Oneadvantage of this embodiment is that more photons are collected persensor in a given period of time, enabling fast acquisition at lowerintegration times given the same photonic flux. Another advantage ofthis embodiment is that the interpolation from the resulting highresolution multiplexed image is improved for some applications.

As one other example of the kinds of filters that can be used inaccordance with an embodiment of the present invention directed at widefield of view imaging, the filters 110, 120, 130 and 140 can be taken tobe 4 different spatial filters that measure weighted linear combinationsof the portion of the scene impinging on the 4 pixels, in 4 differentsectors of a wide field of view image. In this way the techniquesdisclosed herein will allow for the substantial reconstruction of awider field-of-view image.

FIG. 5 shows a system diagram of a device in accordance with the presentinvention, which is comprised of a sensor array 510, a mosaic filter 520(shown in FIG. 5 as 4 parts collectively numbered 520, to indicate that520 is comprised of a mosaic of filters), such as but not limited to thefilters shown in more detail in FIGS. 1-4, and a data acquisition andprocessing module 530. Light from a scene is imaged through the mosaicfilter 520, to the sensor array 510. The output of the sensor array 510is acquired and process by the module 530 in accordance with the presentinvention.

FIG. 6 shows a flowchart for a process in accordance with an embodimentof the present invention, for filling in data from the output of amosaic sensor in accordance with the present invention. The acquisitionand processing module 530 interpolates the mosaic-acquired-data as afirst approximation I in step 610. This is comprised of, for example,extending each spectral filter measured by all pixels that have measuredthe given spectral filter, by interpolating, including but not limitedto using a bilinear interpolation, to fill in values for this missingfilter at the other pixels where the given filter has not been measured,thereby obtaining a first estimate of the true value of the filterresponse at each pixel of the image. An inquiry is made in step 620 todetermine if each pixel for each filter has been measured andapproximated. If the inquiry at step 620 is answered in the affirmative,the acquisition and processing module 530 outputs I₂.

If the inquiry at step 620 is answered in the negative, the acquisitionand processing module 530 proceeds and repeats steps 630 and 640 foreach pixel “x” and for each spectral filter “b.” The acquisition andprocessing module 530 selects the set of pixels for which “b” has beenmeasured, and for which, in addition, the first approximation fullspectral vector is close to that of “x” (that is I(the pixel) is closeto I(x)) in step 630. Then in step 640, the acquisition and processingmodule 530 obtains a second approximation to the true value of aspectral filter value “b” at “x” by interpolating from the values ofthat band “b” for the pixels selected in step 630.

Turning now to the details of a mathematical computation in accordancewith an embodiment of the present invention, these compriseinterpolating a sub-sampled vector-valued function to a fully sampledreconstruction, wherein the interpolation includes a way, for eachpixel, to combine known information about other pixels of the samechemistry (in as much as spectral information corresponds to chemistry).An embodiment of the process is shown in FIG. 7, with a parameter εhaving a predetermined value that determines the threshold ofsensitivity of the algorithm, and comprising the steps of:

710: Interpolate the acquired data bilinearly, resulting in an n×m×Lmultispectral datacube I(i, j, k), (where i and j are spatial indices,and k is spectral index).

720: Form the matrix:${{M\left( {i_{1},j_{1},i_{2},j_{2}} \right)} = \frac{{\mathbb{e}}^{{- {{{\overset{\_}{I}{({i_{1},j_{1}})}} - {\overset{\_}{I}{({i_{2},j_{2}})}}}}^{2}}/ɛ}}{\sum\limits_{{i_{3} = {1\ldots\quad n}},{j_{3} = {1\ldots\quad m}}}{\mathbb{e}}^{{- {{{\overset{\_}{I}{({i_{1},j_{1}})}} - {\overset{\_}{I}{({i_{3},j_{3}})}}}}^{2}}/ɛ}}},$

where I(i, j) denotes the vector <I(i, j, 1), I(i, j, 2), . . . , I(i,j, L)>, (that is, with i and j fixed, k ranging over all values from 1 .. . L)

730: For k=1. . . L, calculate the reconstructed multispectral datacubeĨ(i, j, k), as follows:${\overset{\sim}{I}\left( {i,j,k} \right)} = {\sum\limits_{{i_{1} = {1\ldots\quad n}},{j_{1} = {1\ldots\quad m}}}{{M\left( {i,j,i_{1},j_{1}} \right)} \cdot {I\left( {i_{1},j_{1},k} \right)}}}$

Note that the function e^(−x) ² ^(/ε) in the above formulas can bereplaced by any suitable kernel function, including but not limited tothe characteristic function of a neighborhood of 0. That is, one cantake a more general formula in step 720, substantially equivalent to thefollowing:${{M\left( {i_{1},j_{1},i_{2},j_{2}} \right)} = \frac{f\left( {{\overset{\_}{I}\left( {i_{1},j_{1}} \right)},{\overset{\_}{I}\left( {i_{2},j_{2}} \right)},ɛ} \right)}{\sum\limits_{{i_{3} = {1\ldots\quad n}},{j_{3} = {1\ldots\quad m}}}{f\left( {{\overset{\_}{I}\left( {i_{1},j_{1}} \right)},{\overset{\_}{I}\left( {i_{3},j_{3}} \right)},ɛ} \right)}}},$for any non-negative function f taking 2 vectors and one scalararguments, wherein f has a maximum wherever the 2 vector arguments arethe same, and goes to zero as the distance between the two vectorarguments increases to infinity, and, except at the maximum, goes tozero as the scalar argument goes to infinity.

In a variation on the above algorithm, one can take the sum in step 730only over those pixels (i₁, j₁) that are near (i, j). For example,within a fixed square region centered at (i, j). For example, if such asquare has side length 2C+1, then take:${\overset{\sim}{\overset{\sim}{I}}\left( {i,j,k} \right)} = {\sum\limits_{i_{1} = {{i - {C\quad\ldots\quad i} + {C\quad\ldots\quad j_{1}}} = {j - {C\quad\ldots\quad j} + C}}}{{M\left( {i,j,i_{1},j_{1}} \right)} \cdot {I\left( {i_{1},j_{1},k} \right)}}}$

Of course the sum can be taken over an oval region, or any otherpredetermined region. Note that these constitute linear filtering of theoriginal vector-valued image I, but with a filter that depends on I, sothe process is, in general, a non-linear filtering.

Some embodiments of the present invention comprise systems for thenon-linear filtering of vector valued data as disclosed herein, appliedto a variety of applications beyond the use of reconstruction ofspectral data from the multispectral camera systems described herein.One class of examples include the denoising of images and videosequences.

An embodiment for the denoising of an image comprises the followingsteps. Given an image I₁(i, j), associate a vector valued image I(i, j,k), by taking the values of I₁(i, j) in a rectangle around (i, j). Inother words, number the points in, say, a 5×5 rectangle (or 3×3 or 4×4,etc), from, say, 1 to 25 (or 1 . . . 9, or 1 . . . 16, respectively,etc). Then, for each pixel (i, j), calculate I(i, j, k) for k=1 . . . 25by taking the values of I₁ at the corresponding 25 numbered locationsaround (i, j). Next, apply the algorithm disclosed herein to compute Ĩor {tilde over (Ĩ)}, and these will provide denoised versions of theoriginal image. Of course, there are other methods to extent theoriginal image I(i, j) to a vector-valued image I(i, j, k), and thetechnique described extends to these. For datasets including but notlimited to videos, seismic data and hyperspectral image sets, the datais already of the form I(i, j, k) and so, again, the embodimentsdescribed apply as methods to denoise or otherwise compress or enhancethis data. Compression, for example, can be accomplished bydown-sampling this data in a way analogous to the way in which themultispectral cameras described herein sub-sample spectral data. In thegeneral case, this results in a lower-resolution image, that can beprocessed as described herein. The steps of such a processing can bedescribed as the steps of 1) magnification of the image, to produce ahigh resolution image, 2) formation of a vector valued image (e.g. inthe 5×5 rectangle discussion above); 3) computation of Ĩ or {tilde over(Ĩ)} as a reconstructed image.

One skilled in the art of diffusion mathematics will see that thetechniques disclosed herein relate in part to methods for approximateexpanding of functions on the top few diffusion coordinates ofassociated graphs. To this end, the embodiments above describe theapplication of a filtering operation to the original image data I(i, j),or I(i, j, k), in order to produce new image data Ĩ or {tilde over (Ĩ)}.The reader is referred to U.S. patent application Ser. No. 11/165633,Coifman et al., and U.S. patent application Ser. No. 11/230949, Geshwindet al., both of which are incorporated by reference in their entirety.

Instead or in addition, the filtering operation can be applied to otherfunctions of (i, j, k). For example j(i, j, k)=j; or z(i, j, k)=(i, j)(i.e. a function from R³ to R²). For this last function, z, applicationof the algorithms described has applications to the problem of imageregistration. For another example, expanding the 5×5 example disclosedherein to include a series of addition coordinates generated by theaction of say a grand tour path within the linear group, yields analgorithm for rapidly generating so-called fractal encoding coefficientsof the original data.

The diffusion graph of the various spectral measurements (i.e. the graphconsisting of all spectra labeled by the pixel) reorganizes the imageinto clusters of similar pixel spectra with low bandwidth (bandwidthhere is in the sense of information content, and not photonic spectralrange). In this way, an embodiment of the present invention comprisesdetermining the correspondence of the material content or chemistry inthe image across the pixels of the sensor. This ability to organize thesub-sampled spectra enables the identification of the different materialregions or spatial chemical distributions in the scene. The knowledge ofthe distribution of materials in the scene imaged on the sensor is usedto deduce the value of the missing spectral information. We estimate thetrue spectrum of each individual pixel at a given band, by consideringall points in the same cluster neighborhood for which that band isknown, and infer the missing value. That is: we infer unknown spectralvalues for a first pixel from spectral values that are known for thisfirst pixel, but restricting to looking at other pixels of the samechemistry as the first pixel. When each material occurs in sufficientlymany places in the scene the success of this method of spectral imageryis assured.

An embodiment of the present invention comprises a standard color CCDsensor disposed to measure red, green and blue pixels of a color imagein a mosaicked pattern such as those that are standard in the art,including but not limited to the Bayer pattern. In this embodiment, themathematical algorithms described herein provide for an improved methodof demosaicking the images acquired with such standard color cameras.

Another embodiment of the multispectral camera comprises additionally abeam splitter, creating two copies of the input image. Each copy is sentto a different CCD, with potentially different mosaics of wavelength. Inan exemplary embodiment, at least one pixel in each mosaic co-set, forthe two different CCDs, corresponds to the same spectral filter. In thisway, the two images can be registered. This split system allows for,say, twice as many wavelengths per pixel/region, without having tospread occurrences of the same wavelengths further apart.

Some embodiments of the present invention comprise the use of the firstapproximation or second approximation spectral vectors such as thoseobtained by the method in FIG. 6, to track the motion of various pixelsin a scene. This is accomplished by exploiting the fact that thesemultispectral images contain a spectral vector for each pixel. In avideo mode, this results in vector valued video. In the present aspect,motion tracking algorithms are disposed to act on said vector valuedvideo, for enhanced and improve motion tracking.

In an embodiment of the present invention, for each individual locationor spatial resolution element in the image of the scene on the sensor weidentify its true spectrum in a given band by imaging the scene untilthe location is imaged or sensed by a sensor measuring that band. Sincelocations in the scene are tracked in this manner, all spectra oflocations will eventually be recorded if the relation of the camera toscene changes from frame to frame resulting in a spatial shift of theimage on the distributed spectrally discriminating sensors.

While the foregoing has described and illustrated aspects of variousembodiments of the present invention, those skilled in the art willrecognize that alternative components and techniques, and/orcombinations and permutations of the described components andtechniques, can be substituted for, or added to, the embodimentsdescribed herein. It is intended, therefore, that the present inventionnot be defined by the specific embodiments described herein, but ratherby the appended claims, which are intended to be construed in accordancewith the well-settled principles of claim construction, including that:each claim should be given its broadest reasonable interpretationconsistent with the specification; limitations should not be read fromthe specification or drawings into the claims; words in a claim shouldbe given their plain, ordinary, and generic meaning, unless it isreadily apparent from the specification that an unusual meaning wasintended; an absence of the specific words “means for” connotesapplicants' intent not to invoke 35 U.S.C. §112 (6) in construing thelimitation; where the phrase “means for” precedes a data processing ormanipulation “function,” it is intended that the resultingmeans-plus-function element be construed to cover any, and all, computerimplementation(s) of the recited “function”; a claim that contains morethan one computer-implemented means-plus-function element should not beconstrued to require that each means-plus-function element must be astructurally distinct entity (such as a particular piece of hardware orblock of code); rather, such claim should be construed merely to requirethat the overall combination of hardware/firmware/software whichimplements the invention must, as a whole, implement at least thefunction(s) called for by the claim's means-plus-function element(s).

1. A multi-spectral imaging apparatus, comprising: an array of filtersin a mosaic pattern; a sensor array disposed to receive an image thathas been filtered by said array of filters; and an acquisition andprocessing module disposed to receive the output of said sensor arrayand for processing said output of said sensor array to provide aprocessed image, wherein said acquisition and processing module isoperable to: perform a first interpolation on said output of said sensorarray to provide a first approximation; and perform a secondinterpolation on the values of said first approximation to provide asecond approximation by approximating values at each pixel of saidprocessed image by, for each given pixel in said processed image,interpolating over the values from said first approximation evaluated atthose other pixels for which the values of said first approximation atsaid other pixels are close to the values at said given pixel.
 2. Theapparatus of claim 1, wherein said sensor array comprises a plurality ofpixels; wherein said first approximation comprises a vector of valuescomprising values corresponding to each of said filters for each pixelin said plurality of pixels; and wherein said acquisition and processingmodule is operable to perform said second interpolation on the values ofsaid first approximation to provide a second approximation byapproximating values at each pixel of said processed image by, for eachgiven pixel in said processed image, and each given filter,interpolating over the values corresponding to said given filter fromsaid first approximation evaluated at those other pixels for which saidvector of values of said first approximation at said other pixels areclose to said vector of values at said given pixel.
 3. The apparatus ofclaim 2, wherein said array of filters is an array of spectral filters.4. The apparatus of claim 2, wherein said acquisition and processingmodule is operable to detect and track explosives.
 5. The apparatus ofclaim 2, wherein said acquisition and processing module is operable todetect and track biohazards.
 6. The apparatus of claim 2, wherein saidacquisition and processing module is operable to recognize targetautomatically.
 7. The apparatus of claim 2, wherein said acquisition andprocessing module is operable to perform said first interpolation byinterpolating said output of sensor array bilinearly to provide an(n×m×L) multispectral data cube I(i, j, k), where i and j are spatialindices and k is spectral index.
 8. The apparatus of claim 2, whereinsaid acquisition and processing module is operable to perform saidsecond interpolation by: forming a matrix${{M\left( {i_{1},j_{1},i_{2},j_{2}} \right)} = \frac{f\left( {{\overset{\_}{I}\left( {i_{1},j_{1}} \right)},{\overset{\_}{I}\left( {i_{2},j_{2}} \right)},ɛ} \right)}{\sum\limits_{{i_{3} = {1\quad\ldots\quad n}},{j_{3} = {1\quad\ldots\quad m}}}{f\left( {{\overset{\_}{I}\left( {i_{1},j_{1}} \right)},{\overset{\_}{I}\left( {i_{3},j_{3}} \right)},ɛ} \right)}}},$where I(i, j) denotes the vector <I(i, j, 1), I(i, j, 2), . . . , I(i,j, L)>, with i and j fixed, k ranging over all integer values from 1 toL; and calculating a reconstructed multispectral datacube${\overset{\sim}{I}\left( {i,j,k} \right)} = {\sum\limits_{{i_{1} = {1\ldots\quad n}},{j_{1} = {1\ldots\quad m}}}{{M\left( {i,j,i_{1},j_{1}} \right)} \cdot {I\left( {i_{1},j_{1},k} \right)}}}$for k ranging over all integer values from 1 to L.
 9. The apparatus ofclaim 8, wherein said acquisition and processing module is operable toperform said second interpolation by forming a matrix:${{M\left( {i_{1},j_{1},i_{2},j_{2}} \right)} = \frac{{\mathbb{e}}^{{- {{{\overset{\_}{I}{({i_{1},j_{1}})}} - {\overset{\_}{I}{({i_{2},j_{2}})}}}}^{2}}/ɛ}}{\sum\limits_{{i_{3} = {1\ldots\quad n}},{j_{3} = {1\ldots\quad m}}}{\mathbb{e}}^{{- {{{\overset{\_}{I}{({i_{1},j_{1}})}} - {\overset{\_}{I}{({i_{3},j_{3}})}}}}^{2}}/ɛ}}},$where I(i, j) denotes the vector <I(i, j, 1), I(i, j, 2), . . . , I(i,j, L)>, with i and j fixed, k ranging over all integer values from 1 toL.
 10. The apparatus of claim 9, wherein said acquisition and processingmodule is operable to perform said second interpolation by calculating areconstructed multispectral datacube:${\overset{\sim}{\overset{\sim}{I}}\left( {i,j,k} \right)} = {\sum\limits_{i_{1} = {{i - {C\quad\ldots\quad i} + {C\quad\ldots\quad j_{1}}} = {j - {C\quad\ldots\quad j} + C}}}{{M\left( {i,j,i_{1},j_{1}} \right)} \cdot {I\left( {i_{1},j_{1},k} \right)}}}$for a fixed square region centered at (i, j) and having a length of2C+1.
 11. A method of multi-spectral imaging, comprising the steps of:receiving an image that has been filtered by an array of filters in amosaic pattern by a sensor array; and processing said output of saidsensor array to provide a processed image by: performing a firstinterpolation on said output of said sensor array to provide a firstapproximation; and performing a second interpolation on the values ofsaid first approximation to provide a second approximation byapproximating values at each pixel of said processed image by, for eachgiven pixel in said processed image, interpolating over the values fromsaid first approximation evaluated at those other pixels for which thevalues of said first approximation at said other pixels are close to thevalues at said given pixel.
 12. The method of claim 11, wherein saidsensor array comprises a plurality of pixels; and further comprising thesteps of: performing said first interpolation on said output of saidsensor array to provide said first approximation comprising a vector ofvalues comprising values corresponding to each of said filters for eachpixel in said plurality of pixels; and performing said second saidsecond interpolation on the values of said first approximation toprovide a second approximation by approximating values at each pixel ofsaid processed image by, for each given pixel in said processed image,and each given filter, interpolating over the values corresponding tosaid given filter from said first approximation evaluated at those otherpixels for which said vector of values of said first approximation atsaid other pixels are close to said vector of values at said givenpixel.
 13. The method of claim 12, wherein the step of receivingcomprises the step of receiving said image that has been filtered by anarray of spectral filters in a mosaic pattern by a sensor array.
 14. Themethod of claim 12, wherein the step of processing comprises the step ofprocessing said output of said sensor array to detect and trackexplosives.
 15. The method of claim 12, wherein the step of processingcomprises the step of processing said output of said sensor array todetect and track biohazards.
 16. The method of claim 12, wherein thestep of processing comprises the step of processing said output of saidsensor array to recognize target automatically.
 17. The method of claim12, wherein the step of performing said first interpolation comprisesthe step of interpolating said output of sensor array bi-linearly toprovide an (n×m×L) multispectral data cube I(i, j, k), where i and j arespatial indices and k is spectral index.
 18. The method of claim 12,wherein the step of performing said second interpolation comprises thesteps of: forming a matrix${{M\left( {i_{1},j_{1},i_{2},j_{2}} \right)} = \frac{f\left( {{\overset{\_}{I}\left( {i_{1},j_{1}} \right)},{\overset{\_}{I}\left( {i_{2},j_{2}} \right)},ɛ} \right)}{\sum\limits_{{i_{3} = {1\quad\ldots\quad n}},{j_{3} = {1\quad\ldots\quad m}}}{f\left( {{\overset{\_}{I}\left( {i_{1},j_{1}} \right)},{\overset{\_}{I}\left( {i_{3},j_{3}} \right)},ɛ} \right)}}},$where I(i, j) denotes the vector <I(i, j, 1), I(i, j, 2), . . . , I(i,j, L)>, with i and j fixed, k ranging over all integer values from 1 toL; and calculating a reconstructed multispectral datacube${\overset{\sim}{I}\left( {i,j,k} \right)} = {\sum\limits_{{i_{1} = {1\ldots\quad n}},{j_{1} = {1\ldots\quad m}}}{{M\left( {i,j,i_{1},j_{1}} \right)} \cdot {I\left( {i_{1},j_{1},k} \right)}}}$for k ranging over all integer values from 1 to L.
 19. The method ofclaim 18, wherein the step of performing said second interpolationcomprises the steps of forming a matrix:${{M\left( {i_{1},j_{1},i_{2},j_{2}} \right)} = \frac{{\mathbb{e}}^{{- {{{\overset{\_}{I}{({i_{1},j_{1}})}} - {\overset{\_}{I}{({i_{2},j_{2}})}}}}^{2}}/ɛ}}{\sum\limits_{{i_{3} = {1\quad\ldots\quad n}},{j_{3} = {1\quad\ldots\quad m}}}{\mathbb{e}}^{{- {{{\overset{\_}{I}{({i_{1},j_{1}})}} - {\overset{\_}{I}{({i_{3},j_{3}})}}}}^{2}}/ɛ}}},$where I(i, j) denotes the vector <I(i, j, 1), I(i, j, 2), . . . , I(i,j, L)>, with i and j fixed, k ranging over all integer values from 1 toL.
 20. The method of claim 19, wherein the step of performing saidsecond interpolation comprises the step of calculating a reconstructedmultispectral datacube:${\overset{\sim}{\overset{\sim}{I}}\left( {i,j,k} \right)} = {\sum\limits_{i_{1} = {{i - {C\quad\ldots\quad i} + {C.j_{1}}} = {j - {C\quad\ldots\quad j} + C}}}{{M\left( {i,j,i_{1},j_{1}} \right)} \cdot {I\left( {i_{1},j_{1},k} \right)}}}$for a fixed square region centered at (i, j) and having a length of2C+1.